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I.  INTRODUCTION 


The  problem  addressed  herein  is  the  evaluation  of  asymptotic  statistics  for  estimates 
of  autoregressive  (AR)  parameters  from  observations  of  mixed  time  series  of  known  order. 

The  observed  time  series  is  considered  to  be  mixed  if  the  series  is  generated  by  an  autoregres¬ 
sive  moving-average  (ARMA)  process  or  by  an  autoregressive  process  observed  in  additive 
white  noise  (AR  +  N).  It  has  been  shown  by  Walker  [  1  ]  and  Pagano  (21  that  a  time  » 
formed  by  the  addition  of  an  autoregressive  process  of  order  (M)  and  white  noise  shoulu  oe 
modeled  as  an  ARMA  process  of  order  (M,M).  In  both  cases  the  estimation  of  the  AR  param¬ 
eters  and  the  evaluation  of  estimate  statistics  are  complicated  by  the  mixed  nature  of  the 
observed  time  series. 

For  an  AR  process  of  known  order  Mann  and  Wald  (3]  showed  that  the  least-squares 
estimate  of  the  process  parameters  coincides  with  their  maximum  likelihood  estimate.  It  was 
also  proven  in  [3]  that  the  least-squares  parameter  estimate  errors  are  jointly  multivariate 
normal  with  zero  mean  and  finite  covariance  matrix.  The  structure  of  the  asymptotic 
covariance  matrix  was  also  evaluated.  Walker  [  1 1  was  the  first  to  examine  estimating  the 
AR  parameters  for  an  AR  +  N  process.  In  [  1  ]  the  asymptotic  efficiency  and  variance  were 
evaluated  for  a  first  order  AR  process  observed  in  the  presence  of  additive  noise.  For  the 
mixed  ARMA  process,  Gersch  [4]  proved  that  asymptotically  unbiased  esFmates  of  the  AR 
parameters  can  be  obtained  by  using  the  “higher  order”  Yule-Walker  equations.  The  structure 
of  the  asymptotic  covariance  matrix  was  obtained,  but  asymptotic  normality  was  not  proven. 

In  this  report,  it  is  shown  that  the  “higher  order”  Yule-Walker  estimates  of  the  AR 
parameters  for  both  types  of  mixed  processes  are  asymptotically  jointly  multivariate  normal. 
The  structure  of  the  asymptotic  covariance  matrix  is  calculated  for  both  types  of  process. 

The  proof  follows  that  of  Kromer  [5] ,  in  which  a  similar  asymptotic  normality  result  was 
proven  for  an  AR  process  without  noise. 

These  results  extend  the  work  of  Gersch  [4]  in  that  the  estimates  are  shown  to  be 
asymptotically  jointly  multivariate  normal.  The  structure  of  the  asymptotic  covariance 
matrix,  obtained  herein,  is  shown  to  be  equivalent  to  that  obtained  by  Gersch.  The  demon¬ 
stration  of  asymptotic  normality  for  the  AR  parameters  for  an  AR  +  N  process  is  a  new  result. 
The  evaluation  of  the  structure  of  the  asymptotic  covariance  matrix  for  the  AR  +  N  case 
extends  the  result  Walker  [  1  ]  obtained  for  a  first  order  AR  process. 
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II.  PARAMETER  ESTIMATOR  FOR  MIXED  SERIES 

Assume  that  the  observed  discrete  parameter  time  series  { Yt }  is  a  real,  zero-mean, 
stationary,  normal  process.  The  series  ( Y^}  is  generated  by  a  mixed  autoregressive  moving- 
average  process  of  known  order  (M,M) 

Yfal  Yt-1 - aMYt-M=Ttt-Mt-l - bMTjt-M  M 

where  the  sequence  {i7t }  is  assumed  to  be  independent,  identically  distribute  (i.i.d.) 

N(0,  o^).  We  evaluate  the  autocovariance  function  for  this  process  by  multiplying  (1) 
through  by  Yt_k,  and,  taking  expectations  term  by  term,  we  get 


E[Yt_kYtl 


a!  ^tYt_k,  Yt-1l - aME[Yt.kYt_M] 


“ElYt-k^t1  "bl  ElYt-kVl1 - bMEtVk^t-M1 


Since  Yt_k  depends  only  on  inputs  r/t_j  for  t-j  <  t-k,  it  follows  that 


E[  Yt_k 


0  k>j 
rYt?  k 


(2) 


From  (2)  we  see  that  the  range  of  j  is  0  to  M.  Defining  Ry(k)  =  E[  Yt  Yt_k] ,  we  can 
write  (2)  as 

RY(k)  -  aj  Ry(k-l) - aM  Ry(k-M)  =  0  K  =  M+l,  M+2, . .  . ,  2M  (3) 


Define  the  (MXM)  matrix  by 


Ry(M)  Ry(M-l)  .  .  .  Ry(l) 
Ry(M+l)  ,  .  .  Ry(2) 


[ry(2M-1) 
and  the  (MX  1 )  vectors 

AT  =  [aj,  a2,  •  •  • ,  aMl 


Ry(M) 


Rjj  =  [Ry(M+l),  Ry(M+2),  ....  Ry(2M)l 


The  relationships  defined  by  (3)  can  be  written  as 


Ih  A  =  .Eh 


(4) 


\ 


These  relationships  are  usually  referred  to  as  the  “higher  order"  Yule-Walker  equations. 
Define  the  covariance  estimator  by 
N-|k| 

MO-fi-  2  Y,Yt+|k|. 

t=l 

If  the  true  covariances  in  Tpj  and  are  replaced  with  the  corresponding  covariance 
estimates,  the  resulting  matrix  and  vector  are  denoted  by  Tjj  and  fry  and  the  solution  of 

IH  A  =  Rh  (5) 


provides  consistent,  but  not  efficient,  estimates  of  the  autoregressive  parameters  for  a  mixed 
autoregressive  moving-average  time  series  [6] . 

III.  ASYMPTOTIC  STATISTICAL  PROPERTIES 


In  this  section  we  prove  that  if  the  “higher  order”  Y-W  equations  are  used  to  estimate 
the  AR  parameters  of  a  mixed  time  series,  the  estimate  errors  are  asymptotically  jointly  multi- 
variable  normal. 


Define  the  following  vectors: 


=  [Ry(1),  Ry(2),  .  .  . ,  Ry(2M)] 
-  [e*\  e*2\  .  , .  ei2MX] 

=  [e~*^,  . . .  e~i2MX] 


* 


-2M  =  -2M  -2M 

-2M  =  -2M  -2M 

-2M  =  -2M  +  -2M 

=  [0,  0, . . . ,  0] 
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Assumption  I:  { Yt }  is  a  normal,  real,  zero-mean  stationary  time  series  and 

OO 

^  ik  Ry(k)l  <°° 

lC=_oo 


6.  E  Parzen,  Efficient  estimation  of  stationary  time  series  mixed  schemes,  Stanford  U,  Tech  Report 
16  on  contract  Nonr-225(80),  1971 
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Theorem  1:  Let  {Y(}  satisfy  Assumption  I,  then 

N*  |ry(1)-Ry(1)|  ...  .,Nyi  |fcY(2M)-RY(2M)| 

are  asymptotically  jointly  multivariate  normal  with  zero  mean  and  covariance  structure  given 
by 

lim  cov  I  N^a(RY(k)  -  RY(k)>,  (RY(j)  -  RY(j))  I 
N-+<» 

it 

-  2ir  f  +  ^(\)d\|  (6) 

-7T 

The  asymptotic  covariance  matrix  structure  is  given  by 

it 

t  =  J  U2M0y  (X)dX.  (7) 

-it 


Proof:  This  result  follows  c  aectly  from  Theorem  5.2  of  Brillinger  [7] . 
Lemma  1 :  There  exists  a  (MX2M)  matrix  D  such  that 
D  (R2M  -  R2M)  =  Sfl  ~Ih  A 


where 


~aM  "al 

o 

• 

o 

i 

03 

2 

-3!  1  0 

o 

o 

“aM  *  * 

♦  0 
♦  •  0 


Proof:  This  result  follows  directly  fro:.,  tae  definition  of  the  matrix  D  and  the  Y-W  relation¬ 
ship 


EH-rH  £=£• 

4 

Lemma  2:  Let  {Yt }  satisfy  Assumption  I  and  assume  {Y^ }  is  an  ARMA  (M,M)  series. 
The  elements  of  the  vector 

n‘/j  (Rh-£h  A) 


7.  DR  Brillinger,  Asymptotic  properties  of  spectral  estimates  of  second  order,  Biometrics.  56, 2, 
p 375-390, 1969 
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are  asymptotically  jointly  multivariate  normal  with  zero  mean  and  covariance  matrix 
structure 

lim  cov  I N*  (£h  -£h  A).  N*  (£h  -Ih  A) T  |  «  D  * 

N-*°° 

ir 

■  2*r  J  D  U-2M  dX  (9) 

-it 

Proof:  From  Lemma  1  we  have  that 

(&H  “Xh  A)  =  D  (R2M  -  &2M> 

Since  D  is  a  matrix  of  constants,  we  can  conclude  from  Theorem  1  that  the  elements  of 
the  vector 

(&H  -£h  A) 


are  asymptotically  jointly  multivariate  normal  with  zero  mean  and  covariance  ma  trix  structure 
given  by  equation  (9)  ■ 

Lemma  3:  Assume  that  { Yt }  is  an  ARMA  (M,M)  series,  then  we  have  that 

~_1  p  _i 

p  1  — i — *.  r<  1 

— H  N->oo  -H 

p 

where  the  notation  indicates  convergence  as  N-+°o  is  “in  probability.” 

Proof:  Given  the  previous  definition  of  the  (MXM)  matrix  we  have 


Ry(M)Ry(M-1)  •••  Ry(D 

Ry(M+  1 ) 

“*  1<y(2) 

&Y(2M) 

•••  Ry(M) 

thus,  by  Theorem  8B  of  Parzen  [8]  we  can  conclude 


— H  iH  • 


8.  E  Parzen,  An  approach  to  time  series  analysis,  Ann  Math  Stat.  vol  32,  p  951-989,  1961 


5 


Gersch  [4]  proves  that  the  matrix  Tj|  is  nonsingular;  thus, JT7*  exists.  Since  the  matrix 

2  2  n 

inverse  is  a  continuous  function  (from  to  )  on  a  neighborhood  containing  Tj-j, 
it  follows  that 

r"1  r"1  (10) 

— H  N-»°°  ~H 

We  now  state  and  prove  the  main  result  of  this  report.  In  the  course  of  the  following 
development  we  will  use  the  notation 


Coy  |n«$,n*qt|  e 


This  expression  should  be  interpreted  as  implying  that  the  vector  of  random  variables  § 
converses  in  probability,  as  to  a  vector  of  random  variables  that  are  jointly  multivariate 
normally  distributed  with  zero  mean  and  covariance  matrix  ©. 

Theorem  2:  Let  {Yt }  satisfy  Assumption  I  and  assume  ( Yt }  is  an  ARMA  (M,M)  series, 
then  the  AR  parameter  estimate  errors 

‘  N1/j  (aj  -  aj ),  N*  (S2  -  a2) . (fiM  -  aM) 

are  asymptotically  jointly  multivariate  normal  with  zero  mean  and  covariance  matrix  structure 
given  by 


cov  N1/l(A-A),  N*(A-A)T 


N-*« 


n 

2w  J  I'1  DU2MDT(C^)T  4(X)dX 


Proof:  Define  the  (M X 1 )  vector  Z  by 


=  N*4 (RH  -  rH  A)  -  r->  (RH  - £h  a) 


By  the  results  of  Lemma  2  and  Lemma  3  we  can  conclude  that 


z=N*<r-'  -r-1)  (EH-rHA)  t, 
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Let  (ft, S',  P)  be  the  underlying  probability  space.  For  arbitrary  e  >  0  and  N  >  M  define 
A^n*  |  wefli:  |Zj  jjl < e i  *  1 ,  2, . . . . M | 

* 

By  ( 1 2)  we  have  that  for  every  tjejO,  1 )  there  exists  a  N  such  that 
P(AeiN)>l-„  N  >  N*  ^  . 

But  since  |Zj  j^l  <  e,  for  i  =  1,  2, . . . ,  M,  this  implies  the  existence  of  the  vector 
Ai  (Ih~  f  H  A)  f°r  ^  40  c  ^€,N“  Th*s  anc*  (5)  imply  that  we  can  write 

(a-a)=Ih1(£h-£hA) 

for  all  to  e  Ae  Substituting  this  i'esult  into  the  definition  of  the  vector JZ  we  get 

Z  =  (A  -  A)  -  n^Ih  (Eh  ~Xh  A) 

for  all  to  e  Af  jyj.  Since  the  selection  of  e  and  rj  is  arbitrary  we  can  conclude  that 

N* (£- A) -N* -£HA)  ^  4m  t>3> 

Therefore,  applying  Lemma  2,  the  stated  result  follows. 

■ 

We  have  proven  that,  when  the  higher  order  Yule-Walker  equations  are  used  to 
estimate  the  AR  parameters  of  a  mixed  autoregressive  moving-average  process,  the  parameter 
estimation  errors  are  asymptotically  jointly  multivariate  normal  with  mean  zero  and  covariance 
matrix  structure  given  by 

cov  j  (A  -  A),  N*4  (A  -  A)T  | 
ir 

2n  J  r^j  D  U2M  gT  (C^1)1  *2  (X)  dX 

-w 

Let  us  now  examine  the  form  of  a  typical  element  of  this  covariance  matrix,  but  first  look  at 
a  typical  element  of  D  U2j^  D^,  call  it  ^ j.  We  have 


2M  2M 

Skj  2  2  dkn  Unm  djm 
n-1  m-1 

2M  2M 

*22  dkn  djm  (ei(n"m)X  +  e‘(n+m)X) 


n-1 

m-1 

2M 

2M 

2M  2M 

2 

2  dkr 

i  djm 

ei(n-m)X  + 

22 

di,  d'  ei(n+m)X 
ukn  ajm  e 

n-1 

m-1 

n=l  m-1 

2M 

2M 

2M 

.  x  2M 

2 

dkneinX 

2 

h.  ,,-imX 
ajm  e 

+  2d|tn 

einX  imX 

e  z  jm 

n=l 

m-1 

n-1 

m-1 

Examining  each  term  independently,  using  the  definition  of  the  matrix  and  defining 


M 

A(e“iX)  =  1  - 

aj  exp^~*JX\  we  get 

j=l 

2M 

K+M-l 

2 

dkn  ein^  = 2  aM+k-n  e*n  v  +  ei(M+k)x 

n-1 

n-k 

=  A(e~iX)  eKM+k)X 


2M  j+M+1 

2  djfn  e~'m^  =  ”  2  aM+j-m  e'imX  e-i<M+i» 

m-1  m=j 

=  A(eiX)  e~'(M+j)X 
We  can  now  express  ^  j  as 

$kj  =  A(e_iX)  A(eiX)  +  A(e“iX)  A(e“iX)  e^k+i)^  ei2MX 

Define  the  matrix  P  by 


7T 

E  =  2tt J  DU2MDT  4(X)dX. 
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a  typical  element  of  P,  call  it  is  given  by 


7T 

fkJ - *  J  «kj  4<X)dX 


IT 

=  2ir  J*  A(e“iX)  A(eiX)  e1*1^  tf>2  (X)  dX 
-rr 

ir 

+  2*  f  A(e-»)  A(e-»)  ei0‘+i»  *i2MX  4 <X>  dX 


At  this  point  we  state  and  prove  the  following  Lemma. 

Lemma  4:  Let  {Yt}  satisfy  Assumption  I  and  assume  {Yt}  is  generated 
by  a  mixed  autoregressive  moving-average  process,  then 


7T 

2jt  J  (A(e~iX))2  ei(k+J)X  ei2MX  02  (X)  dX  «  0 


Proof:  From  the  definition  of  the  power  spectral  density  lor  an  ARMA  process  we  can 
write 

^  *»  lA(e-‘x)l2 

M  M 

where  B(e“‘X)  =  1  -  ^  bg  exp(-i£X)  and  A(e"‘X)  =  1  -  ^  an  exp(-inX). 

2=1  n=l 


pkJ  =  2n  J  (A(e“iX))2  ei<k+j>X  ei2MX  tf2  (k)  dX 
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then  after  the  substitution  for  <j>y(\)  given  above  we  have 


4  l 


*  _  J7.  [ 

?kj  “  2n  J 


[B(eiX)  B(e~iX)] 
(A(eiX))2 


exp  j  i(k+j  +  2M)X  dX  . 


We  can  write 


oo  oo 


(1-  2  an exP(_inX))”2  =  2  2  Cn.  Cn2  exp  |‘(n]+n2)x| 


nl=0  n2=0 


and  with  bQ  =  1 


[B(eiX)  B(e~iX)]  2  =  ***  2  \  bC2  \  %  C*P  | i(C |  * 


® 1  =0  ®4=0 


Now,  we  have 


M  oo  oo 


,*.4"-  f  2  ••  2  2  2 

kj  2tt  7  C1=0  e4=0  n,  =0  n2=0 


b£,  bC2  bC3  bC4  Cn,  Cn2 


exp  Ji[(k+j+2m)  +  (fij-fi2+g3-£4)  +  (nj  +  n2)]  xj  dX 


,4  M 


OO  OO 


2 


c,=0  £4=0 


b£j  b£2  b£3b£4 


/  2  2 

4  n,  =0  n2=0 


C  C 

nl  n2  ’ 


•exp  ji  [(nj+n2)  +  (k+j+2M+£j-£2+f  ^-£4)]  xj  dX 
From  equation  (3.56)  of  Kromer  (5]  we  have  that 


00  00 


/  2  2 

4  n,=0  n2=0 


cn,  Cn2  exp 


i  (n  j  +n->+d)  X  j  dX  =  0ford>0 


thus  it  follows  that  j  =0  since 


k  +  j  +  2M  +  £1  -  £2  +  £3-£4>0 


By  the  result  of  Lemma  4  we  have  that 


t 

pkj  *2»  J  A(e*iX)  A(eiX)  e^k-i>X  (X)  dX  .  (15) 

nr 

We  will  now  examine  the  form  of  this  covariance  term  for  two  cases:  ( 1 )  the  observed 
process  (  Yt }  is  a  mixed  autoregressive  moving-average  ARMA  process  and  (2)  { Yt }  is  the 

sum  of  an  autoregressive  process  plus  white  noise  AR  +  N. 

1.  AUTOREGRESSIVE  MOVING-AVERAGE  CASE 

We  have  previously  stated  that  the  power  spectral  density  for  the  ARMA  process  is 
given  by 


_  o~  B(eiX)  B(e“iX) 


2jt  A(e‘x)  A(e“‘X) 


Substituting  this  expression  into  ( 1 5)  we  get 
ir 

„  I 

V 


*>kj  =°2 


II 

j  B(eiX)B(e-‘X)ei(k-j)X  0Y(x>dx  • 


(16) 


Define  0V(X)  to  be  the  power  spectral  density  for  the  moving-average  component  of  the 
process;  ie, 

0V(X)  =  o*  B(eiX)  B(e~iX)  . 

Now  we  can  rewrite  ( 1 6)  as 


7T 

^kj  =  j  Vk)^Y<x)e'(k'j)XdX 


M 


=  2  rv<E)*yfi“k+j) 
e=-M 


(17) 
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where,  by  definition,  rv(6)  = 

M-8 

2  bjbj+8  and  Ry(fi)  =  E  [Y^  Yt_g] 
j=o 

We  can  write  (17)  as 

M 

pkJ  =  rv(0)  RY(k-j)  +  ^  rv(e>  |RY(E-k+i)  +  RY  <’e~k+j)  |  • 

6-1 

Define  the  matrix  r®  element  by  element  as  r®(k-j)  =  Ry  (2-k+j)  and  P  is  the  matrix  of 
elements  pkJ,  then  we  have 

M 

P  »  rv(0)  r°+  2  rv<C>  |l*  +  <C£)T  I  • 

6=1 

Thus,  our  final  result  follows  from  (11), 

Cov  |n‘/2(A-A),  N1/j  (A-A)T|  Ch  ~  (CH)T  ’ 


which  is  equivalent  to  the  result  obtained  by  Gersch  (4) .  Gersch  did  not  prove  asymptotic 
normality,  but  the  forms  of  the  asymptotic  covariance  matrices  are  equivalent. 

2.  AUTOREGRESSIVE  PLUS  NOISE  CASE 


It  was  shown  by  Walker  [  1  ]  and  Pagano  [2]  that  when  the  observed  time  series  is 
generated  by  an  AR  process  of  order  M  plus  white  noise  an  ARMA  (M,M)  model  as  given  by 
(1 )  can  be  used  to  represent  the  series.  If  we  let 

Yt  =  Xt  +  nt 

where  Xt  is  an  AR  process  of  order  M  and  nt  is  i  j.d.  N(0,  a^)  then  ( Yt }  can  be  represented 
by  (1)  and  the  general  covariance  relationship  of  (1 1)  applies.  A  more  specific  relationship, 
that  takes  into  account  the  properties  of  the  AR  +  N  series,  is  calculated  below. 


We  can  write  the  power  spectral  density  for  the  AR  +  N  process  as 


#y(X)  -  + 


A(eiX)  A(e“*x) 


I 


Substituting  this  expression  into  (IS),  we  get 


4  * ' 

Pk j  =  ~  J  A(eiX)  A(e_iX)  e'<k->>X  dX 


~°n“  V 
2ir 


it  * 

f  ei(k-j)XdX  +  0f2  J  ^X(X)  ei(k"j>X  dX 
zL  -v 


4 

=  I  A(e‘X)  A(e“‘X)  e*^k”^X  dX  +  2on2  o 62  6(k-j)  +  oe2  Rx(k-j) 


o  ^  r 

=  on~  oc2  8(k-j)  +  o62  RY(k-j)+^L  /  A(eix)  A(e-iX)  e^^dX 

2ir  J 

-IT 


For  the  AR+N  process  we  can  express  the  matrix  p  as 

n 


P  =  0n2  022  I+0e2  r0+^G._  /  A(eiX)  A(e"iX)  GMQM* 

2ir  J 


dX 


-7T 


and  the  Final  result  is 


cov  N1/2  (A  -  A),  N1/?  (A  -  A)T  |  JT“ 1  P (T, " 1  )T. 


IV.  CONCLUSIONS 

The  estimation  of  the  AR  parameters  for  mixed  time  series  has  been  studied  for  two 
types  of  mixed  seiies-time  series  generated  by  an  ARMA  process  and  time  series  generated 
by  an  AR  process  observed  in  white  noise.  In  both  cases  the  estimates  have  been  shown  to  be 
asymptotically  jointly  multivariate  normal  with  zero  mean  and  covariance  matrix  structure 
given  by 

cov  |n,/2(a-a),n'/j(a-a)|  rH_1  pcr^1)1 

where  the  form  of  the  matrix  Pis  dependent  on  the  type  of  mixed  series  assumed. 

The  results  obtained  relative  to  the  asymptotic  statistics  associated  with  the  AR+N 
time  series  are  of  considerable  interest  to  researchers  working  in  the  area  of  AR  spectral 
estimation  of  noisy  signals. 
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